Marker-based augmented reality authoring tools

ABSTRACT

An augmented reality-based content authoring tool is presented. A content author arranges machine-recognizable markers in a physical environment. A computing device operating as the authoring tool recognizes the markers and their arrangement based on a captured digital representation of the physical environment. Once recognized, augmented reality primitives corresponding to the markers can be bound together via their primitive interfaces to give rise to a content set. The individual primitives and content set are instantiated based on the nature of the marker&#39;s arrangement.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/225,928, filed Apr. 8, 2021, now U.S. Pat. No. 11,270,516 issued Mar.8, 2022, which is a continuation of U.S. application Ser. No.16/579,685, filed Sep. 23, 2019, now U.S. Pat. No. 10,997,790 issued May4, 2021, which is a continuation of U.S. application Ser. No.15/846,041, filed Dec. 18, 2017, now U.S. Pat. No. 10,424,123 issuedSep. 24, 2019, which is a continuation of U.S. application Ser. No.14/851,897, filed Sep. 11, 2015, now U.S. Pat. No. 9,892,560 issued Feb.13, 2018 which relates to and claims the benefit of U.S. ProvisionalApplication No. 62/049,351 filed Sep. 11, 2014 and entitled“MARKER-BASED AUGMENTED REALITY AUTHORING TOOLS”, the entire disclosuresof which are hereby wholly incorporated by reference.

STATEMENT RE: FEDERALLY SPONSORED RESEARCH/DEVELOPMENT

Not Applicable

BACKGROUND

Technical Field: The technical field of the present disclosure isaugmented reality content development technologies.

The background description includes information that may be useful inunderstanding the present inventive subject matter. It is not anadmission that any of the information provided herein is prior art orrelevant to the presently claimed invention, or that any publicationspecifically or implicitly referenced is prior art.

Consumers have access to a broad spectrum of augmented reality (AR)experiences via their personal electronic devices. Consider the appWikitude®. Wikitude provides information about landmarks bysuperimposing informational content about a landmark on a cell phonedisplay. As a user points the camera of their cell phone toward alandmark, the cell phone identifies the landmarks in the field of viewbased on the cell phone's geo-location. In response to the viewingdirection and location information, Wikitude returns informationalpop-ups that can be superimposed on images of the landmarks.

Another example of an AR experience includes an interior design tooloffered by IKEA® and developed by Adornably™. A user places an IKEAcatalog on a floor with the catalog opened to a page having an image ofa furniture piece of interest. The user then uses their cell phone,enabled with the Adornably app, to capture an image of the room, floor,and catalog page. The app then superimposes a 3D computer generatedmodel of the furniture piece of interest over an image of the room andanchored to the catalog's position on the floor (see URLwww.fastcodesign.com/3024113/this-crazy-app-makes-a-magazine-an-interior-design-tool).Yet another example of AR-based interior design includes those describedby international patent application publication WO 2014/048497 to Wanget al. titled “Method of Image Processing for an Augmented RealityApplication”, filed Sep. 28, 2012.

Still another interesting AR experience includes a toy constructionsystem described by international patent application publication WO2012/160055 to Muthyala et al. titled “A Toy Construction System forAugmented Reality”, filed May 22, 2012. Muthyala describes LEGO®-styleconstruction blocks that physically couple with a computer-recognizablemarker. A device such as a computer can then display content relative tothe marker on a display screen.

AR experiences available to consumers can be quite complex andinteractive. In order to provide such rich experiences, contentdevelopers require complex development environments to create theexperiences in the first place. Even simple AR experiences can requirearcane tools or a significant amount of effort by a developer. Further,existing development tools are beyond the ability of a non-technicalconsumer to create their own experiences.

There are numerous AR content development tools available. For example,U.S. patent application publication 2010/0164990 to Van Doorn titled“System, Apparatus, and Method for Augmented Reality Glasses forEnd-User Programming”, filed internationally on Aug. 15, 2006, describesan environment where an end-user programmer is able to visualize anambient intelligent environment. Still, the end user must be technicallyastute in order to create desirable content. Another complex authoringtool is described in European patent application EP 2 490 182 to Ooi etal. titled “Authoring of Augmented Reality”, filed Feb. 6, 2012. In theOoi approach, an author must navigate complex menus and interfaces inorder to create their desired content.

Further effort has been applied to toward providing access to ARcontent. For example, U.S. Pat. No. 8,682,879 to Kim et al. titled“Marker Search System for Augmented Reality Service”, filed Apr. 16,2010, describes a system where a search engine returns markers inresponse to a key word search. The markers can be identified by a deviceand, in turn, the device can obtain web content based on the marker. Thecontent, an image for example, can then be superimposed on a display ofthe device. Although useful in accessing AR services, Kim still fails toprovide insight into how a layperson can generate their own AR contentwith ease.

Even further progress is made by Berry et al. as described in the paper“The Augmented Composer Project: The Music Table” (Proceedings of theSecond IEEE and ACM International Symposium on Mixed and AugmentedReality (ISMAR 2003)). Berry describes a system where users are able tolearn about music through an AR system. The user creates patterns ofcards and the AR system converts the patterns into musical phrases whilealso superimposing AR content on images of the cards. Unfortunately, theapproach taken by Berry only leverages individual pieces of music boundto a card. Further the Berry approach does not provide for interactivityamong the AR content. For example, the musical patterns generated by thecards are merely individual pieces of music played as the same timerather than inter-connectable functional elements through which morecomplex content can be created.

Thus there remains a need for AR authoring tools that would allowconsumers to generate AR content in a simplified fashion.

All publications identified herein are incorporated by reference to thesame extent as if each individual publication or patent application werespecifically and individually indicated to be incorporated by reference.Where a definition or use of a term in an incorporated reference isinconsistent or contrary to the definition of that term provided herein,the definition of that term provided herein applies and the definitionof that term in the reference does not apply.

The following description includes information that may be useful inunderstanding the present inventive subject matter. It is not anadmission that any of the information provided herein is prior art orrelevant to the presently claimed inventive subject matter, or that anypublication specifically or implicitly referenced is prior art.

In some embodiments, the numbers expressing quantities of ingredients,properties such as concentration, reaction conditions, and so forth,used to describe and claim certain embodiments of the disclosure are tobe understood as being modified in some instances by the term “about.”Accordingly, in some embodiments, the numerical parameters set forth inthe written description and attached claims are approximations that canvary depending upon the desired properties sought to be obtained by aparticular embodiment. In some embodiments, the numerical parametersshould be construed in light of the number of reported significantdigits and by applying ordinary rounding techniques. Notwithstandingthat the numerical ranges and parameters setting forth the broad scopeof some embodiments of the invention are approximations, the numericalvalues set forth in the specific examples are reported as precisely aspracticable. The numerical values presented in some embodiments of thedisclosure may contain certain errors necessarily resulting from thestandard deviation found in their respective testing measurements.

Unless the context dictates the contrary, all ranges set forth hereinshould be interpreted as being inclusive of their endpoints andopen-ended ranges should be interpreted to include only commerciallypractical values. Similarly, all lists of values should be considered asinclusive of intermediate values unless the context indicates thecontrary.

As used in the description herein and throughout the claims that follow,the meaning of “a,” “an,” and “the” includes plural reference unless thecontext clearly dictates otherwise. Also, as used in the descriptionherein, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise.

The recitation of ranges of values herein is merely intended to serve asa shorthand method of referring individually to each separate valuefalling within the range. Unless otherwise indicated herein, eachindividual value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g. “such as”) provided with respectto certain embodiments herein is intended merely to better illuminatethe invention and does not pose a limitation on the scope of theinvention otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element essential to thepractice of the various embodiments of the present disclosure.

Groupings of alternative elements or embodiments disclosed herein arenot to be construed as limitations. Each group member can be referred toand claimed individually or in any combination with other members of thegroup or other elements found herein. One or more members of a group canbe included in, or deleted from, a group for reasons of convenienceand/or patentability. When any such inclusion or deletion occurs, thespecification is herein deemed to contain the group as modified thusfulfilling the written description of all Markush groups used in theappended claims.

BRIEF SUMMARY

The inventive subject matter provides apparatus, systems and methods inwhich a one can leverage an augmented reality (AR) authoring tool tocreate content based on using real-world objects. One aspect of theinventive subject matter includes an embodiment of an authoring tool, acomputing device for example, comprising a recognition engine and acontent creation engine. The recognition engine includes one or morecomputing modules that configure the recognition engine to identifyknown markers based on detected signatures from a digital representationof a physical environment. The recognition engine can be configured toor programmed to receive a digital representation of a set of markerswithin the physical environment, where the digital representation couldinclude image data, video data, or other types of digital data. Therecognition engine continues processing the digital data by identifyingone or more markers from the digital representation. The recognitionengine further uses identities of the markers as recognized from thedigital representation to identify inter-connectable AR objectprimitives where the AR object primitives represent portions of contentthat can be computer generated (e.g., graphics, solids, game elements,etc.). For example, the identities of the markers (e.g., GUIDs, indices,etc.) can be used to query a primitives database for corresponding ARobject primitives. In some embodiments, each marker can have one or morecorresponding AR object primitives. Further, each AR object primitivescan include a primitive interface through which it can couple to otherAR primitives. The content creation engine also comprises a collectionof modules that configure the content creation engine to build an ARcontent set from the AR object primitives. The content creation enginecan obtain a content creation rules set comprising instructions orcommands that govern how the AR object primitives are to be boundtogether. The content creation engine is further configured orprogrammed to obtain a set of AR content primitives based on orincluding the AR object primitives provided by the recognition engine.The content creation engine, following the content creation rules set,binds the AR content primitives together via their correspondingprimitive interfaces to generate an AR content set. Once the AR contentset is suitably instantiated, the content creation engine can cause adevice, perhaps the tool itself, to present the AR content set on adisplay where the AR content set is superimposed on an image of thephysical environment.

Various objects, features, aspects and advantages of the inventivesubject matter will become more apparent from the following detaileddescription of preferred embodiments, along with the accompanyingdrawing figures in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages of the various embodimentsdisclosed herein will be better understood with respect to the followingdescription and drawings, in which like numbers refer to like partsthroughout, and in which:

FIG. 1 is a schematic overview of a device operating as an augmentedreality content authoring tool.

FIG. 2 presents an example of combining AR content primitives into acomplete content set based on recognized markers.

FIG. 3 presents an example set of real-world markers placed in anarrangement.

FIG. 4 illustrates recognition of the real-world markers in FIG. 3 via ahand-held device.

FIG. 5 illustrates the creation of AR content based on the arrangementof the markers in FIG. 3 .

DETAILED DESCRIPTION

It should be noted that any language directed to a computer should beread to include any suitable combination of computing devices, includingservers, interfaces, systems, databases, agents, peers, engines,controllers, or other types of computing devices operating individuallyor collectively. One should appreciate the computing devices comprise aprocessor configured to execute software instructions stored on atangible, non-transitory computer readable storage medium (e.g., harddrive, solid state drive, RAM, flash, ROM, etc.). The softwareinstructions preferably configure the computing device to provide theroles, responsibilities, or other functionality as discussed below withrespect to the disclosed apparatus. Further, the disclosed technologiescan be embodied as a computer program product that includes anon-transitory computer readable medium storing the softwareinstructions that causes a processor to execute the disclosed steps. Inespecially preferred embodiments, the various servers, systems,databases, or interfaces exchange data using standardized protocols oralgorithms, possibly based on HTTP, HTTPS, AES, public-private keyexchanges, web service APIs, known financial transaction protocols, orother electronic information exchanging methods. Data exchangespreferably are conducted over a packet-switched network, the Internet,LAN, WAN, VPN, or other type of packet switched network.

One should appreciate that the disclosed techniques provide manyadvantageous technical effects including enabling a computing device tooperate as a digital augmented reality (AR) content authoring tool. Thecomputing device is configured or programmed to digitally recognizeknown markers and map the markers to digital AR content primitives. Asan author organizes or arranges the markers in a physical environment,the computing device detects the changes and re-configures the device'smemory, processor, or other computing elements to enable the AR contentprimitives to aggregate together according to the marker arrangement.The aggregate of the AR content primitives gives rise to new AR contentthat can be presented on a digital device's display.

The following discussion provides many example embodiments of theinventive subject matter. Although each embodiment represents a singlecombination of inventive elements, the inventive subject matter isconsidered to include all possible combinations of the disclosedelements. Thus if one embodiment comprises elements A, B, and C, and asecond embodiment comprises elements B and D, then the inventive subjectmatter is also considered to include other remaining combinations of A,B, C, or D, even if not explicitly disclosed.

As used herein, and unless the context dictates otherwise, the term“coupled to” is intended to include both direct coupling (in which twoelements that are coupled to each other contact each other) and indirectcoupling (in which at least one additional element is located betweenthe two elements). Therefore, the terms “coupled to” and “coupled with”are used synonymously. Within the context of a networking environment,the terms “coupled to” and “coupled with” are also used euphemisticallyto describe two or more networked computing devices that arecommunicatively connected with each other over a network, possibly viaone or more intermediary devices.

FIG. 1 illustrates a possible schematic overview of AR content authoringsystem 100 where a computing device is configured or programmed tooperate as content AR authoring tool 110. In the example shown ARauthoring tool 110 is illustrated as a unitary computing device, whichcan comprise a non-transitory computer readable memory (e.g., RAM,FLASH, ROM, HDD, SSD, etc.) and one or more processors (e.g., ASIC,FPGA, multi-core processor, Intel i7®, AMD, ARM, etc.). Softwareinstructions are stored in the memory and configure the deviceprocessor(s) or processor cores, through execution of the softwareinstructions, to fulfill the roles or responsibilities of contentauthoring tool 110 as described herein.

Although content authoring tool 110 is illustrated as a unitary device,it should be appreciated that content authoring tool 110 could also takeon different forms depending on the nature of the authoring environment.In some embodiments, the components of content authorizing tool 110 canbe distributed among other computing devices in AR content authoringsystem 100, possibly distributed over a network 115 (e.g., Internet,WAN, LAN, personal-area-network, VPN, P2P network, ad-hoc network,wireless network, wired network, 802.11, Wi-GIG, Bluetooth, etc.). Forexample, recognition services provided by recognition engine 120 couldbe provided by a recognition server (e.g., HTTP server, cloud server,etc.) via a web-services API. Further, sensor 112 could be a remotecamera located on another computer, perhaps through a video chatapplication (e.g., Skype®, Zoom®, etc.). Thus, content authoring tool110 can comprise a unitary device or a federation of distributedcomputing devices that coordinate their behaviors over network 115.

In convenient embodiments, as illustrated, content authoring tool 110can be integrated into an easily accessible computing device, possiblyincluding a smart phone, cell phone, tablet, phablets, game console ordevice, a camera, robot, augmented reality viewer (e.g., Google® Glass®,Oculus™ Rift™, etc.), a smart watch, or other type of mobile computingdevice. One of the myriad advantages of the disclosed technology is thatit enables AR content authoring on such devices that have beentraditionally extremely difficult to use for computer programming aswell as for AR content creation. For the sake of discussion, theinventive subject matter will be presented from the perspective of atablet device having a software application or “app” that configures orprograms the tablet to operate as content authoring tool 110. Further,the inventive subject matter will be presented from the perspective ofcreating AR content as part of a child's entertainment product.

Content authoring tool 110 comprises several components to facilitateeasy generation of AR content represented by AR content set 134. In theexample shown, content authoring tool 110 includes recognition engine120 and content creation engine 130. Content authoring tool 110 couldalso include one or more of sensor 112 that captures digitalrepresentation 122 of a physical environment having one or more ofmarkers 142 placed into arrangement 140. It is also possible, but notrequired, that content authoring tool 110 could include or incorporateprimitive database 160 or rules set database 150, rather than accessingsuch databases over network 115. Recognition engine 120 represents acomputing recognition module (i.e., hardware and software) thatrecognizes markers 142 and arrangement 140 from digital representation122, and then maps the recognized markers into a collection ofinter-connectable AR object primitives represented by AR primitives 126.Content creation engine 130 represents a computing module that binds ARprimitives 126 into AR content set 134 according to one or more rulesset 132 where AR content set 134 can be rendered for display on display190.

Markers 142 represent real-world, physical items placed in a physicalenvironment. In the example shown, markers 142 are illustrated asplaying cards in order to convey the concept that any type ofrecognizable object could serve as a marker 142. The term “marker” isused euphemistically to represent objects that have been registered toone or more AR primitives. In some embodiments, markers 142 could beelements of a toy, perhaps action figures, building blocks (e.g., LEGO®bricks, wooden blocks, etc.), or other type of toys. Markers 142preferably comprise sufficient digitally detectable features that, whenexisting in one or more modalities of digital data, can be identifiedvia processing algorithms executed on the digital data. For example,markers 142 could comprise custom-made cards having pictures or imagesthat further include a detectable scale invariant set of features (e.g.,SIFT descriptors; see U.S. Pat. No. 6,711,293 to Lowe titled “Method andapparatus for identifying scale invariant features in an image and useof same for locating an object in an image” filed Mar. 23, 2000), thesubstance of which is incorporated herein by reference. Examples ofmarkers 142 could include cards (e.g., trading cards, playing cards,game cards, etc.), 3D objects (e.g., toys, action figures, machineparts, tokens, or other objects having detectable textures, etc.),building blocks, printed matter (e.g., magazine articles, newspaperarticles, printed images, books, credit cards, etc.), electronicdisplays of external devices, such as tablets, smartphones or smartwatches where the various displays can be arranged to create an ARdisplay (e.g. a group of individuals can combine smart watches orsmartphone screens to create a visible content) or other types ofobjects.

Consider a scenario where the set of markers 142 are associated with atoy, perhaps associated with the Pixar® Cars® brand. Each of markers 142could comprise a collectible trading card having one or more images fromthe Cars movies. Further, the cards can be associated with apurpose-built AR app for mobile devices that can enhance the play valueof the cards. For example, each card could be associated with racetrackprimitives (e.g., start, finish, ramp, obstacles, curves, straights,hills, etc.) where the AR app can recognize the cards and bind theirindividual racetrack primitives into a complete racetrack based onarrangement 140 of markers 142.

It should be appreciated that arrangement 140 may be more than a merecollection of markers 142. The physical placement or juxtaposition ofmarkers 142 in arrangement 140 carries meaning with respect to how ARcontent set 134 should be instantiated. Thus, arrangement 140 could beconsidered as instructions that contribute to construction of AR contentset 134, and should be considered as influencing how AR content set 134behaves. The approach of using arrangement 140 to influence or dictatehow AR primitives 126 integrate with each other has many advantages, notthe least of which is enabling a non-technical individual, a child forexample, to quickly re-arrange markers 142 to give rise to a differentversion of or a new version of AR content set 134 without requiring theauthor to be technically or computer savvy. For example, a child couldconstruct a racetrack based on arrangement 140 as a game level to beplayed with friends. Then, the child could completely change arrangement140 (e.g., add cards, remove cards, shift cards around, stack cards,etc.) to generate a completely new racetrack. In the example shown,markers 142 comprise a circular arrangement. However, it should befurther appreciated that arrangement 140 could comprises otherarrangements possibly including a linear arrangement, hierarchicalarrangement, a tree arrangement, a stacked arrangement, a 3Darrangement, or other types of arrangements. In some embodimentsarrangement 140 can also comprise a temporal arrangement where markers142 can shift location, position, orientation, or other configuration intime.

Recognition engine 120 is configured or programmed to receive digitalrepresentation 122 of arrangement 140 of a marker set of markers 142 ina physical environment. Digital representation 122 can be captured byone or more of sensor 112 and can reflect one or more data modalities.In more preferred embodiments, sensor 112 comprises a digital camera(e.g., CCD, CMOS, etc.) that generates image data as digitalrepresentation 122. For example, the image data could include stilldigital images (e.g., BMP, JPG, PNG, etc.), video frames, video data(e.g., MPEG, etc.), or other types of image data. In some embodiments,content authoring tool 110 leverages an integrated sensor 112 to obtaindigital representation 122. In other embodiments, content authoring tool110 can receive digital representation 122, possibly over network 115,from a remote sensing device having sensor 112. Other data modalitiescould also include audio data, kinesthetic data, tactile data, biometricdata in cases where markers 142 comprises living or animate objects,motion data (e.g., acceleration, etc.), or other types of digital datamodalities. It should be appreciated that sensor 112 is a deviceconfigured to capture a version of digital representation 122 that iscomplementary to the detectable features of markers 142 or arrangement140. Thus, depending on the target modality of markers 142, sensor 112could also comprise a biometric sensor, microphone, tactical sensor, GPSsensor, accelerometer, or other type of sensor.

Recognition engine 120 is further configured or programmed to identifyeach of marker 142's identity in the marker set from digitalrepresentation 122. Referencing back to the example of digitalrepresentation 122 comprising image data, recognition engine 120 canexecute one or more implementations of image processing algorithms togenerate features from the image data. One example of an imageprocessing algorithm includes SIFT referenced previously. Recognitionengine 120 can execute an implementation of SIFT on the image data,which results in a set of SIFT descriptors (e.g., 128 byte valuesrepresenting a gradient orientation histogram of an image patch).Alternative or additional image processing algorithms for deriving imagefeatures include, BRISK, SURF, FAST, BRIEF, Harris Corners, Edges,DAISY, GLOH, HOG, EOG, TILT, VLAD, or others known or yet to beinvented. Implementations of each of the algorithms generate datastructures comprising representations of some form of features (e.g.,descriptors, corners, key points, etc.). The features generated by theimplementations of the algorithms can be used to retrieve informationbound to markers 142.

The identity of each of markers 142 can take different forms. Forexample, a marker's identifier could comprise some form of number orvalue (e.g., an index into a table, a GUID, UUID, an address, a set ofdescriptors, etc.). The identification can be determined by mapping thefeatures derived from digital representation 122 to known identifiers ofknown markers 142. For example, in some embodiments, the markeridentifiers can be stored in a tree structure (e.g., kd-tree, spilltree, etc.) that is arranged in memory based a priori known features ofknown markers. When digital representation 122 is analyzed byrecognition engine 120, the resulting features can be compared to theknown features in the tree to determine which known features are nearestneighbors to the observed features. The result set of known features canthen be used to obtain corresponding known identifiers of known markers142. The identifiers of markers 142 can be stored in alternativestructures as well, for example markers 142 could be stored in a remotedatabase where the marker identifiers are indexed based on knownderivable features of the known markers. In some embodiments, the knownfeatures of markers 142 could be the marker identifier. Exampletechniques for recognizing objects at target objects and returningtarget object information such as marker identifiers that could besuitably adapted for use with respect to the inventive subject matterincludes U.S. Pat. Nos. 7,016,532; 8,224,077; 8,224,078; and 8,218,873all to Boncyk et al, each of which are incorporated herein by reference.

Recognition engine 120 is further configured or programmed to identifyat least one inter-connectable AR object primitive for each marker 142in the marker set from each of the marker's identity. In the exampleshown recognition engine 120 compiles a set of marker identifiers andconstructs one or more queries targeting the indexing system ofprimitives database 160. The query could comprise a submission of allthe identifiers, perhaps packaged as XML, or JSON file, sent toprimitives database 160 over network 115. Still in other embodiments,primitives database 160 could comprises an internally stored look uptable (e.g., direct table, hash table, dictionary, etc.) where one ormore of AR primitives 126 are stored. In such a scenario, recognitionengine 120 can simply retrieve desired information from the look uptable. With regard to the inventive subject matter described herein,markers can represent both terminals and non-terminals when usinggrammar definitions in context-free grammars which permits a compactrepresentation of the potentially complex mappings between markerconfigurations and their meanings.

In response to receiving the query of marker identifiers, primitivesdatabase 160 returns a result set having one or more AR primitives 126representing inter-connectable AR object primitives. The result setcould comprise the data objects and their associated content, or couldinclude pointers to other locations over network 115 or in memory ofwhere the objects can be found. For example, the result set mightinclude file names in a local file system where AR primitives 126 can befound on the presentation device.

Primitives database 160 also could take on different forms depending onthe desired experience. In some cases, primitives database 160 might bepurpose-built only having indexed AR primitives 126 that are relevant tothe installed application. Returning to the racetrack example discussedabove, primitives database 160 might include all racetrack primitivesassociated with a collectible card set, and only those racetrackprimitives. Such an approach is advantageous for creating highlyaccurate and specific applications, but may lack versatility. In otherembodiments, primitives database 160 might include a broad spectrum ofprimitives across numerous possible types of applications. However, thebroad spectrum approach provides for high versatility, but can generatefalse positive matches that could detract from the user experience orconsume excessive resources on the presentation device. Still further,the author could construct or populate primitives database 160 bycorrelating observed markers 142 to known primitives. A custom approachallows the author to create a highly personalized interface andexperience.

AR primitives 126 represent basic inter-connectable building blocksthrough which the author can generate a desired AR content set 134. Eachof AR primitives 126 could be considered individual executable moduleshaving code or data that gives rise to AR elements. Each of the ARelements, could be rendered individually, but would likely not be ofmuch interest. Rather, through stitching together AR primitives 126based on markers 142 and their arrangement 140, the author can generatequite complex new AR content sets 134. Consider again the racetrackexample. AR primitives 126 might represent a curve, a straightway, astart, a finish, and obstacles. Each of these elements is uninterestingindividually by themselves. However, through arrangement 140, a childcould construct an infinite number of racetracks, perhaps as a gamelevel for an AR or computer game.

Depending on the nature of the actual embodiment and targetcapabilities, AR primitives 126 could be widely different. For example,AR primitives 126 could comprises game primitives, program codingprimitives (e.g., variable assignments, conditional logic, loops, etc.),chemical element primitives, electronic schematic primitives, physicalprimitives (e.g., simulation primitives), or other types of primitives.Many of these types of AR primitives are discussed in greater detailnear the end of this document with respect to various use-cases. Oncecontent authoring tool 110 has access to AR primitives 126, contentcreation engine 130 can engage with AR primitives 126 to give rise to ARcontent set 134.

In some embodiments to aid in construction of AR content set 134,content creation engine 130 is configured or programmed to obtaincontent creation rules set 132. Rules set 132 represents instructions bywhich or according to which content creation engine 130 binds ARprimitives 126 together. Content creation engine 130 can obtain one ormore of rules set 132 through various techniques. In purpose-builtembodiments, a specific AR application executing on content authoringtool 110 could be a priori provisioned with an application-specificrules set 132 integrated into the application. Still, in other moregeneric embodiments, content authoring tool 110 can leverage ambientdevice information or observed information to construct a querytargeting the indexing schema of content creation rules set database150. For example, content authoring tool 110 could compile a set ofattributes-value pairs representing a current context of the device. Theattribute-value pairs might represent a location, a time, an event, auser preference, a digital right, a purchased license, or othercontextual information. In response to receiving the query, contentcreation rules database 150 returns one or more rules set 132 to contentcreation engine 130. Still further, rules set 132 might be indexedaccording to known features from one or more of markers 142 or theirspecific arrangement 140. Thus, content creation engine 130 can beconfigured to select rules set 132 from multiple rules sets based onrecognizing the set of markers 142 or based on the device context.

Rules set 132 can include various requirements, conditions, parameters,or other factors that facilitates integration of AR primitives 126. Forexample, AR primitives 126 might represent generic AR primitives andrules set 132 could include rules for customizing AR primitives 126 to aspecific use. Consider an example where AR primitives 126 representcastle building structures bound to purchasable princess trading cards.Each AR primitives 126 might represent different parts of a castle; saya parapet, draw bridge, wall, or other type of structure. However, as adefault, all the AR primitives 126 might be rendered in pastel pink. Achild could change their personal preferences to indicate they prefer adifferent rendering color; perhaps blue. In response, rules set 132carries specific instructions to alter the defaults of AR primitives 126to give rise to a blue castle on display 190 rather than a pink castle.Thus, rules set 132 can carry additional content information orprogramming that might not ordinarily be part of AR primitives 126.Still, it is possible, especially in application-specific orpurpose-built applications, for AR primitives 126 to comprise sufficientrules sets 132 for instructing content creation engine 130 on how tobind AR primitives 126 together.

Content creation engine 130 is further configured or programmed toobtain the set of AR primitives 126 where the primitives areinter-connectable among each other. In embodiments as illustrated, ARprimitives 126 are already local to content creation engine 130.However, in other distributed embodiments, content creation engine 130might be required to receive AR primitives 126 over network 115 from aremote device operating as recognition engine 120 or from other devices.Each primitive in the set of AR primitives 126 comprises a least oneprimitive interface through which is it able to interact with otherprimitives. The primitive interfaces could be considered a programmingapplication programming interface (API), among other types ofinterfaces, and is described more fully with respect to FIG. 2 .

With AR primitives 126 in hand and optional rules set 132, contentcreation engine 130 is configured to generate AR content set 134 bycoupling two or more of AR primitives 126 together via their respectiveprimitive interfaces according to content rules set 132 and as afunction of arrangement 140. From a simplified perspective, outputinterfaces of one primitive can couple to the input interfaces of one ormore other primitives. In a more implementation perspectiverepresentations, references or pointers to a primitive's data structuresand elements (e.g., content, code, etc.) can be passed through an API toanother primitive.

It should be appreciated that arrangement 140 directly influences howrules set 132 should be executed on AR primitives 126. For example,arrangement attributes derived from arrangement 140 can represent inputsinto a primitive's interface. More specifically, the relative distancein the physical environment between two of markers 142 could be used toinfluence the interaction between the two corresponding AR primitives126; perhaps representing a coupling force strength in a physicalsimulation for example. Example arrangement attributes that influencethe function by which AR primitives 126 are coupled could includerelative proximity in arrangement 140, relative orientation inarrangement 140, location in arrangement 140, movement in arrangement140, stacking in arrangement 140, temporal changes, or other observablefactors.

In some embodiments, content creation engine 130 couples primitivesbased on a pair-wise juxtaposition of markers 142 in arrangement 140.The pair-wise juxtaposition represent a comparison of relativeproximities, locations, positions, or other arrangement attributes. Apair-wise approach allows content creation engine 130 to determine whichmarkers 142 should be bound together. For example, in the example shownthe King of Diamonds is physically stacked on the Nine of Clubs and theSix of Diamonds. Thus, content creation engine 130, based on rules set132, could bind AR primitives 126 for these three cards together ratherthan binding the King of Diamonds AR primitive with the Queen of ClubsAR primitive for example. It should be further appreciated, that morethan two primitives (e.g., three, four, or more) could be coupledtogether via their respective interfaces.

Once AR content set 134 is sufficiently instantiated, content creationengine 130 can cause a device, likely content authoring tool 110, topresent AR content set 134 on display 190. For example, referring backto the racetrack example, AR content set 134 might be a fullyinstantiated racetrack that can be superimposed on an image of thephysical environment, perhaps even relative to or anchored to images ofmarkers 142 in arrangement 140. One should note that in someembodiments, AR content set 134 is presented on display 190 in real-timerelative to capturing digital representation 122. Thus, content creationengine 130 can be configured to update AR content set 134 based onmovement of markers 142 within the physical environment. Such anapproach is advantageous because it enables non-technical individuals toquickly react to or address undesirable compositions of AR content set134 and then create modifications. In such cases, content creationengine 130 can track positions of markers 142 in real-time as well inorder to determine new arrangement 140 and corresponding AR content sets134.

Although the main examples of the inventive subject matter focus ontoys, it is contemplated that the disclosed techniques can be applied tomany other areas or markets where AR content set 134 takes oncorresponding aspects. For example, within a programming context, ARprimitives 126 could represent programming constructs and AR content set134 represents programming code (e.g., executable scripts, compliablecode, etc.). In a toy or gaming context, AR content set 134 couldinclude video gaming elements (e.g., levels, tracks, AI behaviors, etc.)or even a virtual game board. Still further, in the context of executinga simulation, AR content set 134 might represent a digital model (e.g.,CAD system, architecture, etc.). Even further, AR content set 134 mightbe a simulation (e.g., chemical reactions, gravity simulation, buildingmodeling, etc.) or perhaps an initial state of a simulation possibly aspart of an educational or training environment. As a more concreteexample, AR primitives 126 might represent chemical elements in aparticular arrangement where AR content set 134 then represents achemical compound (e.g., small molecule, Kinase inhibitor, drug, DNA,RNA, etc.). More complex versions of AR content set 134 could comprise acontrol system built from AR primitives 126, perhaps a remote control ofa device or robot for example. Interestingly, AR content set 134 couldalso comprise an error indicator if markers 142 fail to fall within anacceptable arrangement. For example, the error indictor could indicatepossible programming syntax or semantic errors, incompatible ARprimitives 126, recommendations on a possible rearrangement, or othertype of error conditions. Furthermore in the context of AR primitivesrepresenting puzzle pieces, AR content set 134 could indicate where twomarkers 142 fail to properly interconnect.

Yet another interesting example includes using AR content set 134 togive rise to a new AR primitive 126, where AR content set 134 composesone or more new primitives. For example, AR content set 134 mightrepresent an AI behavior for a gaming character. The AI behavior can beregistered to an existing marker 142 or a new marker. This new ARprimitive then can become available as a building block for new, evenmore elaborate AR content sets 134. An astute reader will realize thatAR content set 132 can be saved in memory for future use.

FIG. 2 presents a more detailed example of combining AR contentprimitives into a complete content set based on recognized markers 242.In this example, markers 242 are illustrated as traditional playingcards. Each card has been registered with a corresponding AR primitiverepresented by AR primitive 226A through 226E, collectively referred toas AR primitives 226. It should be appreciated that the cards can beregistered by deriving digital features (e.g., descriptors, values,identifiers, etc.) from digital representations of the cards, thenbinding the digital features to AR primitives 226. Each of AR primitive226 can include data elements that aid in constructing AR content set234; an augmented reality racetrack in this example. Examples of dataelements can include digital graphics, digital sound data, metadata,relevant APIs, implementations of algorithms, or other data that can bestored in memory for rendering or presenting the AR primitives.

AR primitives 226 can be tagged with data elements such as a markeridentifier (e.g., index, GUID, etc.), which allows a recognition engineto identify which of AR primitives 226 are bound to which markers 242.Further, AR primitives 226 can also include one or more configurationparameters that could be adjusted manually or automatically as afunction of the arrangement of markers 242. The parameters dependheavily on the nature of the target use-case, but might includegraphical parameters (e.g., size, shape, color, lighting, albedo, etc.),duration in time, force strength in a simulation, or other typeparameter.

Of specific interest, each of AR primitives 226A through 226E includesat least one corresponding interface 227A through 227E, respectively,and collectively referred to as interfaces 227. In some embodiments,each of AR primitives 226 has its own internal interface 227, perhapsoperating as an API as illustrated. In other embodiments, interfaces 227could be identified via an address (e.g., pointer, URL, etc.) throughwhich other interfaces exchange data with the corresponding AR primitive226. For example, interface 227B might be a web service accessible via aURL where an intermediary interface server brokers communications amongAR primitives 226. Thus, the inventive subject matter is considered toinclude providing AR primitives 226 via one more web services; perhapsas AR as a Service (ARaaS). Still further, as in the racetrack example,interfaces 227 could comprise a graphic interface via which each of ARprimitive 226 is able to graphically integrate with other AR primitives226.

Each of AR primitives 226 also comprises AR content primitive 228 thatrepresents the digital content to be rendered as part of content set234. In this example, the Ace of Hearts maps to AR content primitive228A, which represents a graphical representation of a racetrack startline. The Queen of Spades maps to AR content primitive 228B, whichrepresents a graphic representation of a racetrack finish line. Stillfurther, the Nine of Clubs maps to a graphical representation of a curveas indicated by AR content primitive 228C. The King of Diamonds maps toa hill-type obstacle as shown by AR content primitive 228D. Finally, theSix of Diamonds maps to AR content primitives 228E representing astraightway. Each of these race track primitives, in this embodiment,includes one or more splines. As the primitives are combined, thesplines are combined to form smooth graphical curves.

The content creation engine combines or otherwise integrates AR contentprimitives 228 into AR content set 234 as shown by the graphical flow atthe bottom of FIG. 2 . The content creation engine uses the parametersfrom AR primitives 226 as well as the arrangement of markers 242 todetermine how to aggregates AR content primitives 228 into AR contentset 234. More specifically, each rack track primitive is adjusted ortransformed (e.g., stretched, narrowed, bent, etc.) so that it isintegrated with a neighboring primitive according to one or more contentcreation rules set; spline interpolation for example. Should the trackauthor wish to make a change, the author need only modify thearrangement of markers 242. The AR racetrack in content set 234 can thenbe saved, edited, modified, deleted, or otherwise modified inpreparation for play. Further, content set 234 can be packaged and sentto other electronic devices. For example, the racetrack can be sent toother tablet computers to create a shared racing experience amongmultiple game players.

FIGS. 3-5 presents a real-world example an AR authoring tool built toleverage real-world markers and to generate a virtual racing track for avideo game. FIG. 3 presents an arrangement of eight real-world cardsrepresenting markers. In the example shown, the cards are sized anddimensioned to match traditional playing western playing cards ortrading card; 2.5″×3.5″. Each card has multiple features of interest.

Each of the cards has one or more icons indicating the nature of thecorresponding primitive associated with the cards. The icons as shown inFIG. 3 are located at the top and/or bottom of the cards, but may bepositioned as desired or for effective recognition. Such icons can serveseveral purposes. First, as indicated, the icons can provide a humanunderstandable indication of the nature of the corresponding primitives.In this example, the icons can indicate straight ways, curves, ramps, orother virtual race track primitives. Further, the icons could also beinterpreted by the AR authoring tool to establish a relative position orrelative orientation of cards with respect to other cards, the camera,or with respect to the entire arrangement if desired.

The cards may also include human readable numbers. The numbers can alsoserve several purposes. With respect to human interactions, the numbersallow a user to easily distinguish one card from another. Such anapproach is considered advantageous in embodiments where multiple,similar cards might be present or available for collection. For example,notice card 02 and card 03 have the same icons, although theirbackground is different. The numbers can aid a human, or a machine forthat matter, in quickly distinguishing similar cards from each other. Insome embodiments, such cards might be collectible in the sense that theyare produced according to a rarity policy (e.g., common, uncommon, rare,ultra-rare, chaser cards, unique, etc.). Therefore, having such numbersor other human readable indicia can help to distinguish cards ofdifferent rarity.

Of more specific interest, each card in this example also includesunique background art so that each card can be uniquely identified bythe authoring tool based on executing one or more implementations ofimage processing algorithms on image data representing the arrangement.In this specific example, the background art comprises images ofsections of a dried mud flat, where the resulting pattern of cracks canbe considered to have observable features at different scales (i.e.,distances between the camera and the arrangement), or considered to havea fractal dimensionality for greater scale invariance. It iscontemplated by the disclosed technology that in addition to thefeatures of the markers that are recognizable by the imaging or scanningdevice, that one or more of the markers could have recognizable featuresthat are below the sensitivity or resolution of sensor (i.e. camera) toprovide a basis of determining authenticity of the marker (i.e.counterfeiting prevention).

The author in this example has chosen to place the cards in a loosefigure eight arrangement. Note that card 05 is between cards 00 and 01.Still, a path can be traced from card 00 through card 05 on to card 01.The path continues from card 01 to card 02, then onto cards 03 and 04,which in turn lead to card 05. The path proceeds from card 05 to card06, then finally to card 07, which leads back to card 00.

In FIG. 4 , a user leverages a tablet device configured to operate as anAR authoring tool to create a virtual race track based on thearrangement of the cards from FIG. 3 . FIG. 4 represents a tabletscreen, with a display of the arrangement of cards from FIG. 3 a viewedthrough a camera on board the tablet. The user directs theforward-facing camera of the tablet to the arrangement of cards. Theauthoring tool recognizes each individual card. In the screen as shownin FIG. 4 , virtual authoring tool content is superimposed on thedisplayed images of the cards. The tool content includes dots which canbe color coded, indicating where matching observable features arepresent (e.g., SIFT descriptors, etc.). These features show that theauthoring tool is able to recognize each card individually anddistinguish each card from each other. The tool content also includes aband (which can be color coded) across the middle of the cards. The bandindicates the orientation of the corresponding card, which furtherindicates the resulting orientation of corresponding AR primitives; racetrack segments and associated splines in this case. It should beappreciated that nearly any type of authoring tool content can bepresented to the user to aid in the user creating their final content(e.g., locations, positions, relative placements, error codes,recommendations on moving markers, etc.).

FIG. 5 illustrates a tablet screen showing the resulting virtual racetrack generated by the authoring tool based on the observed arrangementof cards in FIGS. 3 and 4 . The tablet operating as the authoring toolcollected corresponding race track primitives and interfaces themtogether to from a completed race track based on the arrangement of thecards. The author of the race track can, in real-time, shift position ofone or more cards to change the arrangement, which in turn causes thetablet to instantiate a new version of the race track. Thus, theauthoring tool supports real-time modification or creation of newcontent.

The reader is reminded about the previous discussion regarding card 05placed between card 00 and card 01. In the race track shown in FIG. 5 ,rather than creating an intersection among cards 05, 00, and 01, therace track comprises a ramp over the track. The path of the track fromcard 00 to card 01 extends under the ramp created by card 05. Thus, card05 provides an excellent example of how a primitive of one card, a rampor hill in this case, can be interfaced with other primitives (i.e.,cards 04 and 06) while not interfacing with other primitives based onthe arrangement of the markers. As such it can be appreciated that inthe interaction of the AR primitives, rules may be provided thataddress, and provide different interaction results based upon, amongother things, marker orientation relative to another marker, distance toanother marker, potential overlap with another marker and whether thesubject marker is on top or below in the overlap, whether a third ormore markers are in proximity, touching or overlapping. Also, API orinterfaces can also correspond to the geometry of the physical markers.In card-like markers which sides are juxtaposed with other markers. Inthree dimensional markers, entirely different sets of rules may beimplemented taking into account, among other things, stacking,juxtaposition, geometry, positioning, etc.

Although FIGS. 2-5 provides more detail with respect to authoring an ARracetrack, it should be appreciated that the disclosed techniques extendbeyond authoring AR gaming content. Thus, the disclosed techniques ofenabling a non-technical user to author content through an AR interfacegives rise to numerous advantages and potential use-cases. The followingdiscussion describes additional topics that are considered to fallwithin the scope of the inventive subject matter.

In view that the disclosed techniques enable authoring content via an ARauthoring tool in a physical environment, it is also possible to allowreal-world elements to influence the creation of the content. In someembodiments, the recognition engine can be configured or programmed torecognize non-marker objects (e.g., a toy, a person, a building, etc.)that have not been registered as markers. Upon recognition of suchobjects, the recognition engine can obtain object information, which canbe used to influence the creation of the content set according to thecontent creation rules set. As an example, a magazine cover could be apriori ingested into an object recognition system. When the recognitionengine of the authoring tool recognizes the magazine cover, therecognition engine can obtain information about that particular issue ofthe magazine possibly including page count, width, depth, length,advertisers, titles of articles, article content, or other type ofinformation. Returning to the racetrack example, the content creationengine could leverage the object information to enhance the content set.Perhaps the racetrack would be modified to go around the recognizedmagazine, to go over the magazine but avoid the cover illustrations, orsimply create virtual billboards around the racetrack displayingadvertisements from advertisers or displaying content from the magazine.As referenced previously, suitable techniques for recognizing objectsand obtaining object information are described in U.S. Pat. Nos.7,016,532; 8,224,077; 8,224,078; and 8,218,873 all to Boncyk et al.

Another technique for modifying the content set based on observedreal-world objects includes using observed estimated depth information.An estimated depth represents a distance from a sensor to an object in ascene. Some embodiments can leverage depth sensors the can directlymeasure a depth, possibly based on infra-red projected patterns. Exampledepth sensors include Microsoft® Kinect®, Leapmotion®, Primesense®, orother types of depth sensors. Further, it is also possible to estimatedepth information from pixel information from multiple images (e.g.,real-time video frames) without requiring a dedicated depth sensor. Oneexample technique for generating depth estimates from images isdescribed in U.S. patent application 2012/0163672 to McKinnon titled“Depth Estimate Determination, Systems and Methods”, filed Dec. 20,2011, the substance of which is incorporated by reference. When contentcreation engine obtains or receives depth estimates of the physicalenvironment, the AR primitives and the content set can be modified toaccount for the depth information. Consider a scenario where the authoris attempting to create an AR physics simulation, perhaps as part of aneducational tool. The objects in the simulation (e.g., renderedinteracting AR primitives) can interact with the physical environment asa function of the depth estimates.

Still another example of enabling an author to create content sets basedon real-world objects includes incorporating real-world tools into theobservable environment. For example, an architect might include a ruleror an architect's scale into a scene along with markers that are boundto AR primitives that represent building materials or buildingcomponents. The recognition engine can recognized the scale as discussedabove and the content creation engine can use the scale information toensure the content set is presented properly to scale. As the architectre-positions their scale the content set can be re-rendered to reflectthe changes in scale, orientation, location, or other parameterassociated with the scale.

Depending on the nature of the AR primitives, in some embodiments theauthoring tool can accept direct author interactions with the markers asinput to the AR primitives. As the author makes gestures or generatesother types of interactions (e.g., spoken commands, etc.), the contentcreation engine can alter the parameters of one or more primitives. Forexample, an author could physically tap a marker with their finger orhand to select options associated with the corresponding AR primitives(e.g., color, size, shape, etc.). In the context of programming, perhapsa marker represents a loop structure. The program author could speak anumber of iterations for the loop while pointing at the marker. Inresponse, the recognition engine executes one or more implementations ofautomatic speech recognitions (ASRs) algorithm on the audio data todetermine the loop count or condition. The content creation engine theninputs the count or condition into the content set. Other types ofinteractions with markers that could adjust parameters of correspondingAR primitives include real-time interactions, changing locations,changing orientations, flipping the markers, articulating the marker(e.g., changing a pose of an action figure, etc.), or other types ofinteractions. The advantage of such an approach is clear. Through suchsimple or natural interactions, an author can create complex contentsets while lacking a strong technical background.

As discussed previously, the markers could be nearly any type of machinerecognizable object. Although it is possible for the markers to comprisesome form a code (e.g., bar code, QR code, matrix code, high capacitycolor barcode, glyphs, etc.), such an approach is less desirable becausesuch markers would lack human understandable information or lackaesthetics with respect to an application. More preferred markers wouldbe visually pleasing while also be relevant to the core application forwhich they are intended. Consider a trading card game (e.g., Card Wars™,Magic: The Gathering®, etc.). Each card can be registered as a markerand bound to appropriate AR primitives based just on the image content(e.g., printed art work, painting, colors, etc.) of the card itselfrather than unappealing codes. As players play cards in their tableaus,the AR primitives can be combined to present content to each playerbased on the arrangement of the card in the tableaus. Also consider theSkylander Battlecast® card and gaming platform that provides theinteractivity of a tablet with a collectors card. The disclosedtechnology could provide an enhanced gaming experience to this andsimilar platforms by allowing users the ability to not only recognizethe card, but also allow recognition of the placement or arrangement ofsuch cards in the visual queue of the electronic device employing thedisclosed technology to provide further AR content than what iscurrently recognized. Likewise, the disclosed technology may enhanceother gaming platforms such as PlayStation® Wonderbook® or similarplatforms that employ AR content in a storybook type setting byproviding markers that could be placed and recognized within the currentplatform to provide additional AR content.

The markers can be quite dynamic. In some embodiment, the markers canchange their observable features with time in a manner that causeschanges to the corresponding AR primitives, or even final content set.For example, a box, perhaps constructed like a large six sided die,might have electronic displays on each face (e.g., LCD display, e-paper,e-ink, etc.). The image on each face could change with time where eachimage might be considered a different marker or a different version ofthe same marker. Additional examples of time varying markers couldinclude images presented on the display of a cell phone, a game device,or even a television screen. In some embodiments, television programmingcontent could become a marker, perhaps where the corresponding ARprimitives might include digital models of characters, scene props,advertisements, or other types of AR primitive content associated withthe television programming.

An interesting aspect of the disclosed inventive subject matter is thatthe recognition engine should be able to recognize the markers in amanner that is substantially invariant to scale, at least to withinpractical limits of the sensors. For example, the recognition engineshould be able to recognize an image-based marker if the marker subtendsa large fraction of the field of view of the sensing camera (i.e., closeto the camera), or if the marker subtends a very small fraction of thecamera (i.e., far from the camera). In such cases, the markers can beconstructed to include features that are observable as differencescales. In some embodiments, a marker generator can be configured toprint out (e.g., 2D printer, 3D printer, etc.) markers having thenecessary scale invariance. Interestingly, while some markers can begenerated to have self-similarity at different scales to ensure properrecognition, perhaps based on a fractal dimensionality, it is alsopossible for markers to have dis-similar feature across multiple scalesin a frequency space (e.g., some highs, some lows, and some mid-levels).The approach of constructing markers having dis-similar features atdifferent scales allows a marker to be bound to different AR primitivesbased on scale or depth in a captured image. Returning to the racetrackexample, a marker could be bound to AR primitives representing scenery.If the marker is at the rear of an arrangement from the point of view ofa capturing camera, the observed features at this scale might indicatethat the content set should integrate a set of virtual trees behind thetrack. If the marker is in the middle of the arrangement and therecognition engine observes mid-level features, the content set couldgenerate a virtual billboard. If the marker is near the capturing cameraand has observable fine grained features, the corresponding ARprimitives might be virtual spectators.

The disclosed techniques also enable a broad spectrum of use-cases. Asdiscussed above, content authors are able to create complex video gamesthrough integration of AR primitives based on their arrangement.Further, the approach opens opportunities for extending revenue of videogame properties by creating collectible markers that enable fans tocreate new content for the games. Fans can purchase the collectiblemarkers to unlock AR primitives and then create new levels for the videogames through quick arrangement of their markers in a physicalenvironment. The levels could be shared through video game forums, sold,licensed, or otherwise distributed as desired. As such, one potentialobjective is to create demand for new markers to unlock further ARcontent that can be used in games and other settings. In addition, theAR primitives can be updated for a particular marker, potentiallycreating additional secondary markets for content or as a loyaltyreward. For example, a consumer could pay for different levels ofcontent for the same marker, or the AR content related to a particularmarker may change depending on a level reached in a game, or based uponcustomer loyalty to a particular brand or product line. Also the ARcontent displayed for markers could be dynamic so that changes indisplay can give rise to changing unlocked AR content.

Another use-case includes creating virtual or augmented board games. Themarkers can comprise game elements (e.g., cards, tokens, boards, dice,etc.). Many board games have complex rules that could be difficult tofollow or tedious to manage during play. Through construction of boardgames or construction of a board game setup, the authoring tool or otherdevice can take on the responsibilities for rules enforcement, or forvirtually managing fiddly or tedious gaming components (e.g., chits,tokens, etc.). In one example, one or more arrangements of board gamepieces form one or more AR sets. Also certain arrangements of board gamepieces could unlock information that could be used in the physical gameitself to provide advantages, or detriments to players.

There are a vast number of possible educational use-cases as well aspractical use-cases. As discussed previously, markers can be bound to ARprimitives representing programming modules or lines of code. Forexample, some effort in the past has been directed to simplify learningprograming through presentation of displayed graphical icons; see theScratch™ programming language developed by MIT (see URLscratch.mit.edu). Although Scratch is moderately accessible to childrenor other non-technical users, it still requires significant effort toconstruct programs. In the disclosed approach, rather than requiring acomputer interface, the novice programmer can quickly rearrange markersto alter their program. Such an approach is considered advantageousbecause the learner would be engaged kinesthetically as well asvisually. Further, Scratch only presents programming blocks in a 2Ddisplay. In the disclosed approach AR primitives of programming blockscan be arranged in a 3D environment, which gives rise to interestingcapabilities, especially for visualization of multi-threadedapplications or visualization of multi-dimensional data analysis.Similarly, in creating educational or business presentations, markersrepresenting content could be placed in a desired arrangement in aphysical environment and displayed in the form of slide decks providinga finished presentation to be displayed. Also, particular text can beassigned to markers, and a user could use an arrangement markers tocreate a document based upon the arrangement. For example in draftingcontracts, a user could lay out certain markers representing boilerplatetext, and the disclosed technology could generate display document. Itappreciated that text assigned markers could be useful in creating anynumber of documents employing standard text.

Additional educational uses could include generating simulations forvarious purposes. In some embodiments that focus on electronics, themarkers could correspond to AR primitives that represent electronicschematic primitives. Example electronic schematic primitives couldinclude traces, power sources, diodes, transistors, resistors,inductors, grounds, capacitors, chips, amplifiers, gates, inverters,junctions, switches, connectors, meters, or other types of electronics.The corresponding content set could include a simulation of electricityflowing through a circuit that corresponds to the arrangement of themarkers. The simulation of the circuit can be changed throughmodification of the arrangement of the markers, which allows the studentto develop an intuition for how electronic circuits operate. Othereducational uses include creating physics simulations (e.g., kinematics,gravitation, quantum mechanics, optics, EM fields, wave guides, antennadesign, etc.) where students, or even engineers, can arrange markerscorresponding to physical objects and then create simulations of theobjects interacting.

In a more practical setting, the disclosed techniques could also be usedin conjunction with design of new chemical compounds; drugs for example.The markers can be constructed to represent elements or compounds wherethe corresponding AR primitives represent 3D virtual constructs. Basedon the arrangement of the markers in 2D space or 3D space, the contentcreation engine can illustrate various simulated strengths of bonds(e.g., pi-bonds, covalent bonds, ionic bonds, Van Der Waals forces,etc.) among the primitives. In a complex example, markers couldrepresent very complex proteins. Such an approach allows a researcher toattempt to identify docking sites through intuitive manual manipulationrather than through exhaustive computational modeling. The content sentcan generate an AR overlay indicating a calculated coupling strength.

Yet more uses could be quite varied. The techniques could be used tocreate a virtual model train system where track and train cars could bemodeled based on markers. A choreographer could simulate a dance usingmarkers that represent dances. In a similar vein a sports coach couldsimulate player positions or game plays using markers and associated ARprimitives representing players. Additionally, as mentioned previously,markers could be used to generate control system interfaces for devicesof equipment. The interface designer could explore the efficiency of adesign quickly by simply changing corresponding markers. Also, themarkers could be used in real world maintenance of machinery or assemblyof products. In maintenance of machines for example, the disclosedtechnology could use markers to show where real-world objects fit onother real world objects due to proximity. In an unassembled toy orpiece of furniture sold to consumers the disclosed technology could be atool to be used as an assembly aid, for example the pieces to beassembled (or stickers applied to the pieces) could act as the markersand the AR content could demonstrate whether two pieces are intended tojoined together or provide other vital information in helping with theconstruction of the product. Also, the markers of the disclosedtechnology can be employed with productivity software. For example,cards representing tasks could be laid out in a particular arrangement,such in order of proposed completion or over a calendar, and thetechnology could display AR content to assist the user in planning andefficient completion of tasks.

The markers could also be used as keys to unlock content based on theirarrangement. Such a use-case allows for securing information as well asproviding entertainment value. As an example, an arrangement of markerscould unlock a virtual landscape, with each landscape feature tied to aparticular marker or card, and the arrangement and positions creatingtransitions between landscape features. Such a landscape use could beused in a gaming environment for creating virtual cities, battlefields,playing fields, golf courses, among others uses. Also, a landscape usecould be used in a more practical real world setting such as designingresidential or commercial landscaping. A landscape architecture usecould involve the design and layout of a golf courses, zoos and buildingcomplexes as some examples. Also, markers, such as cards could be usedto lay out events for the temporary location of tables and seating suchas social events (i.e. seating, table arrangement), objects or booths tobe viewed or visited, such as art exhibits, convention halls. Cardmarkers used with the disclosed technology could also be useful asvisual aids in chemistry illustrations, educational demonstrations andphysics demonstrations. Returning the theme of games, collectibleobjects (e.g., figures, toys, trading cards, etc.) could be bound to ARprimitives that are a priori unknown to the author. As the author shiftscorresponding markers around in various arrangements, the AR primitivesmight reveal themselves so that the author discovers the content throughexperimentation. More complex content can then be created and madeavailable to others; thereby, give rise to a shared experience. Also,the disclosed technology can be used with collectible objects as a meansof indexing. For example a user could arrange collectable objects andthe disclosed technology could display a virtual museum providinginformation about piece in the collection, and to also potentiallydemonstrate the completeness of the collection, and to display to theuser, where certain pieces are missing. Also the disclosed technologycould on one hand catalog baseball cards for example, but also provideAR content based upon a particular arrangement of baseball cards. Forexample, completing a circular arrangement of the defensive infield ofthe 1979 Pittsburgh Pirates® from baseball cards could display thestarting pitching rotation for the 1979 World Series® or provide othersimilar AR content.

Another interesting advantage to the disclosed approach is that themarkers point to AR primitives. Thus, as desired, the AR primitivescould be changed so that the same marker or sets of markers could pointto newly created content. In a very real sense, the AR primitives couldbe considered as a type of multi-media programming where the markers orarrangement of markers become a broadcast station for the programming.Also, the disclosed technology could be useful in the marketing areawhere products themselves become the markers (or stickers or objectsattached to the products) and marketers could define and re-define theproperties of the AR primitives. For example, placing two or morebottled beverages from a well-known brand in an arrangement could berecognized the disclosed technology displaying AR content that could beused or leveraged as a reward. For example, the product arrangementcould unlock a code that could be used for coupons or other rewards. TheAR primitives could be changed and modified such that markers andarrangement of markers would only be recognized for a period of time(i.e. limited time offer). Also, the technology could be implemented inuser based level editing where user subscribers (such as through awebsite subscription) can define properties of the AR primitives. Also,in crowd-building level editing, the users can define the AR primitivesof the markers which could provide possible positions where markerscould connect to each other.

It should be apparent to those skilled in the art that many moremodifications besides those already described are possible withoutdeparting from the inventive concepts herein. The inventive subjectmatter, therefore, is not to be restricted except in the spirit of theappended claims. Moreover, in interpreting both the specification andthe claims, all terms should be interpreted in the broadest possiblemanner consistent with the context. In particular, the terms “comprises”and “comprising” should be interpreted as referring to elements,components, or steps in a non-exclusive manner, indicating that thereferenced elements, components, or steps may be present, or utilized,or combined with other elements, components, or steps that are notexpressly referenced. Where the specification claims refers to at leastone of something selected from the group consisting of A, B, C . . . andN, the text should be interpreted as requiring only one element from thegroup, not A plus N, or B plus N, etc.

The above description is given by way of example, and not limitation.Given the above disclosure, one skilled in the art could devisevariations that are within the scope and spirit of the disclosedinventive subject matter. Further, the various features of theembodiments disclosed herein can be used alone, or in varyingcombinations with each other and are not intended to be limited to thespecific combination described herein. Thus the scope of the claims arenot intended to be limited by the illustrated embodiments.

What is claimed is:
 1. A computer-based digital content server system,comprising: at least one non-transitory computer readable memory storingsoftware instructions; and at least one processor coupled with the atleast one non-transitory computer readable memory and coupled with astored indexed digital content set and rules sets, the at least oneprocessor, upon execution of the software instructions, performingoperations comprising: registering a physical, real-world object with atleast one portion of the digital content set by indexing the at leastone portion of the digital content set with at least one digitallyderivable feature of the physical, real-world object; receiving a queryfrom a device over a network, the query comprising the at least onedigitally derivable feature of the physical, real-world object;retrieving, based on the query, the at least one portion of the digitalcontent set and a corresponding rules set governing how the at least oneportion of the digital content set is bound in the digital content set;and causing the device to render the at least one portion of the digitalcontent set as part of the digital content set according to thecorresponding rules set.
 2. The system of claim 1, wherein the physical,real-world object comprises a real-world collectible object.
 3. Thesystem of claim 2, wherein the physical, real-world collectible objectcomprises rarity relative to other real-world collectible objects. 4.The system of claim 3, wherein the rarity is based on a rarity policy.5. The system of claim 1, wherein the at least one digitally derivablefeature comprises at least one image descriptor.
 6. The system of claim1, wherein the query comprises a digital representation of a physicalenvironment.
 7. The system of claim 6, wherein the at least onedigitally derivable feature is derived from the digital representationof the physical environment.
 8. The system of claim 6, wherein thephysical environment comprises the physical, real-world object.
 9. Thesystem of claim 1, wherein the at least one portion of the digitalcontent set is retrievable based on a hash.
 10. The system of claim 9,wherein the hash is based on the at least one digitally derivablefeature.
 11. The system of claim 1, wherein the at least one portion ofthe digital content set is unique.
 12. The system of claim 1, whereinthe at least one portion of the digital content set comprises anexecutable module.
 13. The system of claim 1, wherein the at least oneportion of the digital content set comprises at least one of thefollowing types of digital content: digital graphics, digital sounds,metadata, application program interfaces, and implementations ofalgorithms.
 14. The system of claim 1, wherein the at least one portionof the digital content set comprises a digital content primitive. 15.The system of claim 14, wherein the digital content primitive comprisesa digital augmented reality (AR) content primitive.
 16. The system ofclaim 15, wherein the digital content primitive comprises a gameprimitive.
 17. The system of claim 1, wherein the at least one portionof the digital content set comprises an interface to at least one otherportion of the digital content set according to the rules set.
 18. Thesystem of claim 1, wherein the digital content set comprises anapplication-specific content set.
 19. The system of claim 1, wherein theat least one portion of the digital content set relates to a purchase.20. The system of claim 1, wherein the query comprises a location of thedevice.
 21. The system of claim 20, wherein one or both of the at leastone portion of the digital content set and the rules set are retrievedbased on the location of the device.